Papers with Key-Value (KV)
UNComp: Can Matrix Entropy Uncover Sparsity? — A Compressor Design from an Uncertainty-Aware Perspective (2025.emnlp-main)
Copied to clipboard
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, Lingpeng Kong, Ngai Wong
| Challenge: | Deploying large language models (LLMs) for long-context inference remains challenging due to their substantial memory and computational demands. |
| Approach: | They propose an uncertainty-aware framework that leverages truncated matrix entropy to identify areas of low information content. |
| Outcome: | The proposed framework reduces the KV cache size to 4.74% of the original and achieves a 6% speedup. |
FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference (2025.findings-emnlp)
Copied to clipboard
Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang
| Challenge: | Key-Value (KV) cache reading latency increases with context lengths hindering LLM inference . important tokens are sparsely distributed across the long context, making existing retrieval inaccurate . |
| Approach: | They propose a method to retain a small fraction of KV cache based on token importance . important tokens are often sparsely distributed across the long context . |
| Outcome: | The proposed method reduces decoding latency by 1.2 to 1.5. |
SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Excessive compression during the prefill phase impairs comprehension of reasoning tasks . SCOPE is a framework that performs KV cache optimization during the decoding and prefill phases . |
| Approach: | They propose a framework that performs optimization during the prefill and decoding phases . they propose enabling a sliding strategy to select essential heavy hitters for the decoding phase . |
| Outcome: | Experiments show that SCOPE can optimize key-value cache for long-context generation tasks . the framework can preserve essential information while minimizing memory usage and transfer . |
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression approaches prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Approach: | They propose a structure-aware KV cache compression framework that prioritizes tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Outcome: | The proposed framework preserves long-range dependencies and retrieval robustness. |
GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Efficient Key-Value (KV) cache management is essential for processing long text sequences in large language models (LLMs). |
| Approach: | They propose a graph-based framework that redefines token selection for KV cache compression. |
| Outcome: | The proposed framework can be used in existing KV cache eviction methods such as SnapKV and PyramidKV in a plug-and-play manner. |
Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression (2026.acl-long)
Copied to clipboard
Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin
| Challenge: | Recent methods to reduce the KV cache size fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss. |
| Approach: | They propose an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant. |
| Outcome: | The proposed method can maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting. |
Anchoring the Cache: Mitigating Contextual Hallucination in KV-Compressed Long-Context Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that KV cache compression can increase hallucination scores in LLMs . modern LLM models support extremely long sequences, but their impact on model hallucinosity remains underexplored. |
| Approach: | They propose a decoding-phase strategy that selectively removes generated KV pairs from retrieval heads responsible for retrieving critical information from source context. |
| Outcome: | The proposed method reduces hallucination across multiple models and datasets while preserving computational efficiency. |